A Uniform Convergence Bound for the Area Under the ROC Curve 1

The area under the ROC curve (AUC) has beenadvocated as an evaluation criterion for the bi-partite ranking problem. We study uniform con-vergence properties of the AUC; in particular, wederive a distribution-free uniform convergencebound for the AUC which serves to bound theexpected accuracy of a learned ranking functionin terms of its empirical AUC on the training se-quence from which it is learned. Our bound is ex-pressed in terms of a new set of combinatorial pa-rameters that we term the bipartite rank-shattercoefficients; these play the same role in our resultas do the standard VC-dimension related shattercoefficients (also known as the growth function)in uniform convergence results for the classifica-tion error rate. A comparison of our result witha recent uniform convergence result derived byFreund et al. [9] for a quantity closely related tothe AUC shows that the bound provided by ourresult can be considerably tighter.